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I. INTRODUCTION 


A. WHY MEASURE PERFORMANCE ? 

It is often true that measurement data is collected and then nothing is done with the 
data. If measurement data isn’t used for something, then collecting it is a waste of time, 
effort, and resources. There must be a reason for collecting performance information 
about Intermediate Maintenance Activities. 

Perigord writes, 

"The purpose of measurement is to find evidence of any variation from the 
negotiated specifications. The role of measurement is to identify possible 
improvements and to indicate where to initiate action on failures. Above all, 
measurement should be a potent vehicle of success." [Ref. 1] 
"Success" is definitely a goal of Intermediate Maintenance Activities (IMAs). As 
elements of the United States Navy, they must perform their mission well at all times to 
ensure military readiness. 

The determination that an IMA is performing its mission well is accomplished 
through measuring its performance. Performing "well" can mean many things, including 
performing effectively and efficiently. 

Changes in the military due to the collapse of the Soviet Union and other world 
events may require that some ship maintenance practices be altered. In 1990, U. S. 


Pacific Fleet Staff members agreed that the following assumptions regarding future 


Pacific Fleet maintenance trends were valid: 


® Less resources available for maintenance, 

@ Shorter repair and modernization periods, 

® More technically complex platforms, 

@ Infrastructure changes, such as base closures and consolidation of facilities, 
® Growth of Intermediate level workload, 

® Greater customer awareness of quality, and 


@ Increased emphasis on measurement.[Ref. 2] 


Meaningful measurement data for Intermediate Maintenance Activity performance 
is Clearly desirable. However, it has been and continues to be difficult to obtain. 
Intermediate Maintenance Activities IMAs) came into being in a rather hit or miss 
kind of way. The IMAs were created in their earliest form as a solution to a manpower 
sea duty and shore duty rotation problem that resulted from the end of the United States’ 
involvement in the Viet Nam War. Their maintenance mission was defined after the fact. 
The IMAs were not created in response to an existing maintenance need. No formal 
performance measurement methods were defined. 
Several observers have commented on the methods of performance evaluation that 
were developed informally. Christopher Moe wrote in 1985, 
“At SIMA Norfolk first line managers evaluate shop productivity on professional 
“gut feeling". Characteristics that support this judgmental estimation focus 
generally on personal motivation of shop personnel, skill acquired through on-the- 
job experience, and training received through technical schools. Most of these 
Same managers based considerable weight on their individual ability to gauge 


productivity through visual evaluation of shop activity. This visual perception was 
further supported by each manager’s knowledge of jobs in progress and projected 


completion dates required to meet the ship’s departure from the repair availability." 
[Ref. 3] 


William Marshall gave a simpler description. He listed the following methods used at 


a Shore Intermediate Maintenance Activity (SIMA) for productivity measurement: 


@ Management by walking around, 


® Observation of the parking lot technique, i.e. if the parking lot is empty early in 
the day not enough work is being done, 


® Completion rate, 

@® Number of productive manhours expended, 
@® Amount of shop overtime, 

@ Rework percentage, and 


@ Waterfront reputation.[Ref. 4] 


Obviously, these are subjective measurement methods, difficult to record and analyze. 
More quantitative, repeatable techniques are needed to obtain meaningful measurement 
data. Such data are needed to identify possible improvements and to identify and correct 


failures. 


B. THESIS OVERVIEW 
This thesis is organized into seven chapters. This introduction is Chapter I. 
Chapter II is a description of Navy ship maintenance. Chapter III contains a discussion 


of various terms used in performance measurement. Current Intermediate Maintenance 


Activity performance measures are examined in Chapters IV and V. Chapter VI 
proposes a new performance measure and Chapter VII contains conclusions and 


recommendations. 


Il. NAVY SHIP MAINTENANCE 


A. BACKGROUND 

Ships are complex structures that require constant care and upkeep. They operate 
at sea, a harsh and unforgiving environment; an uncared for ship will quickly deteriorate 
and fail its user. Modern warships have the additional complexity of advanced weapons 
systems and gas turbine propulsion plants. Such Navy vessels must be able to respond 
to crises at any time in addition to performing routine deployments and exercises. There 
is no place for unkempt, broken down ships in the United States Navy. To help ensure 
that the ships of the fleet will be ready when called upon, the Navy has an extensive ship 
maintenance program. 

The Navy ship maintenance program is designed to keep ships at "an adequate 
level of material condition to maximize their required operational availability to the Fleet 
Commanders" [Ref. 5]. In other words, the goal of the maintenance program 
is to keep everything in the ships of the Navy in proper working order and in proper 
condition as much as possible. “Downtime” for any system or component is to be 
minimized. 

The Navy ship maintenance program has three levels, each requiring a different 
degree of capability. The levels are organizational, intermediate, and depot. Each level 


is defined below. 


1. Organizational Level 
The first level of maintenance is the organizational level consisting of the ship 
itself and the sailors on board the ship. Organizational level maintenance is that 
corrective and preventive maintenance accomplished by the ship’s crew. The work isa 
blend of equipment operation, condition monitoring, planned maintenance actions and 
repair ranging from simple equipment lubrication to component change out and, in some 


cases, complete rework in place. [Ref. 6] 


2. Intermediate Level 

The second level of maintenance is the intermediate level consisting of 
Tenders, Repair Ships, Shore Intermediate Maintenance Activities (SIMAs) and Naval 
Reserve Maintenance Facilities (SIMA NRMFs). At these commands Navy personnel 
with specialized facilities and training accomplish intermediate level repair work. 

Intermediate level maintenance is that maintenance which is normally 
performed by Navy personnel stationed on tenders, repair ships, and at SIMAs and SIMA 
NRMFs. It normally consists of calibration; repair or replacement of damaged or 
unserviceable parts, components or assemblies; the emergency manufacture of 


unavailable parts; and providing technical assistance. [Ref. 5] 


3. Depot Level 
Depot level maintenance is that type of maintenance generally requiring a 
greater industrial capability than possessed by either organizational or intermediate level 


activities. It consists of that maintenance performed by shipyards, either private or 


Navy, Naval Ship Repair Activities, or other shore based activities, on equipment 
requiring major overhaul or complete rebuild of parts, assemblies, subassemblies, end 
items, and complete platforms, including manufacture of parts. 

The only work to be scheduled for accomplishment by depot level 
maintenance activities will be that which is not feasible to be accomplished by 
organizational or intermediate level maintenance activities because of insufficient time 
Or manpower, or because it is beyond the capabilities of these fleet maintenance 
activities, or is of such a nature that split responsibility between Fleet and depot 


maintenance activities should be avoided. [Ref. 5] 


B. INTERMEDIATE MAINTENANCE ACTIVITY MISSIONS 

This thesis examines performance measures for the maintenance activities at the 
intermediate level. It is not concerned with organizational or depot level maintenance. 
A further explanation of the missions of Shore Intermediate Maintenance Activities, 
Repair Ships, and Tenders follows. 


The mission of the Shore Intermediate Maintenance Activities (SIMAs) is to: 


@ Perform intermediate level maintenance for ships, 


@ Provide meaningful assignments ashore to support sea/shore rotation in order to 
retain the skilled Petty Officers needed to man the Fleet in peacetime and 
mobilization, 


@ Provide in rate training and experience for sea intensive enlisted personnel who 
repair and maintain shipboard systems, 


@ Provide a mobilization option for wartime maintenance and battle damage repair, 
and 


@ Provide billets co-located with Naval Reserve Force ships to support TAR 

personnel sea/shore rotation and retention. [Ref. 7] 

Like Shore IMAs, the mission of Repair Ships and Tenders is to perform 
intermediate level maintenance for ships. However, their mission does not include the 
manpower rotation considerations and Naval Reserve support concerns listed above for 
Shore IMAs. Unlike Shore IMAs, Tenders and Repair Ships are not fixed in one 
location. They have the mobility necessary to provide intermediate level support, repair 
of battle damage and other emergent repairs to advance and forward areas when required. 
This unique feature, mobility, is also needed to redeploy intermediate maintenance 
capability between theaters in balance with the movement of operating forces. This 


reduces the investment needed for fixed overseas sites. [Ref. 7] 


C. U.S. PACIFIC FLEET INTERMEDIATE MAINTENANCE ACTIVITIES 


1. Description 

There are eleven ship Intermediate Maintenance Activities (MAs) assigned 
to the United States Pacific Fleet. Six are shore based facilities, four are Tenders, and 
one is a Repair Ship. They vary widely in size and capability. Table 1 lists the eleven 
IMAs and the number of personnel assigned to each. 

All IMAs have the capability to do routine intermediate level work such as 
welding, valve work, pump repair, sheetmetal work, and electrical repair. The larger 
IMAs can provide such things as foundry work, diving services, lifeboat maintenance, 


electroplating, sonar repair, and gas turbine engine repair. 


TABLE 1 


U.S. PACIFIC FLEET 
INTERMEDIATE MAINTENANCE ACTIVITIES 


of 3/92 
789 
A 182 


aaa eal ee 
Coronado, CA 

138 
USS Samuel Gompers (AD-37) Alanicain CA 792° 
[USS Jason (AR) | Sam Diego 


"IMA personnel only. Does not include other ship personnel. 













San Diego 


2. Chain of Command 
All Intermediate Maintenance Activities report to the Commander, Naval 
Surface Force Pacific (COMNAVSURFPAC) for the performance of their duties. The 
senior staff officer concerned is the Assistant Chief of Staff, Maintenance and 


Engineering, organizational code N4. Subordinate to N4 is the Force Intermediate 


Maintenance Officer, code N41. A key assistant to N41 is the Intermediate Maintenance 
Plans and Programs Officer, code N411. The many duties of N411 include: 
@ Provide data for budget input and budget requirements and prepare budget 
recommendations for IMAs, 


@ Review Force IMA _ requirements and provide appropriate inputs to 
COMNAVSURFPAC long-term procurement requests, 


@ Participate as the COMNAVSURFPAC representative in development efforts in the 
ADP community which have a direct effect on the quality and/or improvement of 
Force maintenance, 


@ Direct the scheduling of IMA availabilities for Force units, and 


@ Research and monitor IMA utilization performance.[Ref. 8] 


The Force Intermediate Maintenance Officer (N41) and the Assistant Chief of Staff, 
Maintenance and Engineering (N4) review and approve all reports from N411 before they 
are forwarded to higher authority. 

This thesis addresses the final item on the list, the duty to "research and 
monitor IMA utilization performance". It does so by evaluating current methods used 
to monitor IMA performance and by recommending additional methods. Valid 
performance measures are necessary to support recommendations made about IMAs while 
carrying out the other N411 duties listed. 

There is another level in the chain of command between 
COMNAVSURFPAC and the IMAs. The immediate senior for Intermediate 
Maintenance Activities (IMAs) is the local screening authority. The screening authority 


is discussed further in the subsection about work request flow. 
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D. INTERNAL INTERMEDIATE MAINTENANCE ACTIVITY 

ORGANIZATION 

Intermediate Maintenance Activities are job shop facilities. They consist of several 
workcenters (or shops), each organized to do a particular type of work. Table 2 lists 
representative workcenters. Not ail IMAs possess all workcenters. For example, SIMA 
San Francisco has 39 different workcenters while SIMA San Diego has 65. The 
alphanumeric names are standardized; for example, 35A will always refer to an optical 
workcenter regardless of where that workcenter is located. In addition to the production 
oriented workcenters, support workcenters exist as well. These include Planning and 
Estimating, Supply, Quality Assurance, and Automated Recordkeeping and Reports 


(ARRS). 


E. WORK REQUEST FLOW 

A request for intermediate level work can be initiated by a ship for several reasons. 
A piece of equipment may break in such a way that the ship’s personnel cannot repair 
it. Routine tests may show that an item needs refurbishment available only at the 
intermediate level. A workspace may need new storage lockers. Mandatory periodic 
intermediate level tasks may be due. Regular wear and tear may require the replacement 
of various deck fittings, lengths of pipe, or portions of deckplate. 

All requests are submitted via a standard work request document with formatted 
fields of data to enable computer systems to aid in the processing of the requests. Each 


ship maintains a list of its pending work requests, known as the Consolidated Ships 
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TABLE 2 


PARTIAL LISTING OF 
INTERMEDIATE MAINTENANCE ACTIVITY 
WORKCENTERS 











shipfiter 
sheetmetal 
welding | 564 | ripe 
inside machine 
valve repair 
316 
oan 67A 
354 





lagging 
= pump shop key and lock 
gas turbine 


electronics 
| 35k | optical 





Maintenance Plan (CSMP). The list is maintained in a computer format. 

All ships in a given geographic area routinely submit their worklists (CSMPs) to 
the local screening authority. The screening authority is a representative of 
COMNAVSURFPAC responsible for allocating work to the IMAs in that geographic 
area. For example, the screening authority in San Diego assigns work to SIMA San 
Diego, USS Acadia (AD-42), USS Cape Cod (AD-43), and USS Jason (AR-8). When 
deployed overseas, Tenders and Repair ships are under the authority of various deployed 
commanders. 

The screening authority is a work request broker. It reviews all work requests in 
the geographic area for appropriateness and correctness and places the work with a 


maintenance facility that can complete the work. Much of the work is placed with Navy 
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Intermediate Maintenance Activities, although the screening authority also places work 
with civilian contractors. 

Requests for work fall into two categories, planned and emergent. Planned work 
is accomplished during availabilities, marrying the customer ships with the repair 
activity, which can be an IMA or private sector contractor. An "availability " is a period 
of time when the ship is made available for work by the scheduling authority and when 
it receives priority service at the IMA. Availabilities are planned for weeks ahead of 
time to ensure materials are available and all jobs are planned. Current Pacific Fleet 
policy is that each ship should have at least one three-week availability each quarter. 
Mission degrading emergent work is accomplished at any time and is assigned to repair 
activities based on their capability, current workload, and operational necessity. 

After a work request is accepted by the IMA, the job must be planned. Personnel 
from the IMA’s Planning and Estimating workcenter review the job, visit the customer 
ship if necessary to gather additional information, order materials, and plan the number 
of manhours and skill types to get the job done. A lead workcenter is assigned and 
support workcenters are designated. Conducting centralized planning allows the planners 
to load the workcenters with the appropriate amount of work and to prepare realistic 
estimates of job completion dates. The work requests are completed by the IMA 
personnel, the job is accepted as complete and satisfactory by the customer ship, and the 


paperwork is completed to clear that work request from the ship’s worklist (CSMP). 
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Ht. PERFORMANCE MEASURES 


This chapter will define and discuss some terms and ideas critical to an 
understanding of performance measures. First, the idea of measurement itself will be 
examined, Second, various categories of performance measurement will be considered, 


including quantity, efficiency, effectiveness, and productivity. 


A. MEASUREMENT 

"Measure" is defined in the dictionary as "the act, process, or result of determining 
the dimensions, capacity, or amount of something" [Ref. 9]. Moore defines it 
as follows: 


"To measure a property means to assign numbers to units as a way of representing 
that property [Ref. 10]." 


Useful measures require clear definitions of the property to be measured and the 
unit to be used to measure that property. 

Some properties and units are widely accepted and understood. For example, it is 
difficult to find disagreement about what "length" is and difficult to find anyone who says 
"feet" or "meters" are inappropriate units to use to measure length. Unfortunately, there 
is not the same agreement about what to measure as an indicator of performance and 
what units to use for such measurement. Some common ground must be agreed upon 


before any meaningful discourse on performance can take place. 
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B. CATEGORIES OF PERFORMANCE MEASURES 

Performance measures can be categorized in many ways. Considering it via the 
three categories of quantity, efficiency, and effectiveness is useful because the data 
examined later in the thesis in Chapters IV, V, and VI is grouped in that way. Attempts 
to measure productivity requires special comment because productivity can be defined 


using both an efficiency concept and an effectiveness concept. 


1. Quantity 

Quantity is the easiest category to discuss, but is not as easy to apply as it 
seems. Counting the number of items of interest is the straightforward way to find out 
how many of the items exist. If "number of items" is used as a comparison between 
groups, there is an underlying assumption that the items being counted are of the same 
quality. This is not necessarily a valid comparison. All football teams field eleven 
players, but no one would claim that, because they have the same number of players, the 
teams perform equally well. A real estate agent who sells twenty homes a month might 
appear to be a better agent than one who only sells one house every six months, but if 
the one house sold is a multi-million dollar mansion while the twenty others are small, 
low-priced homes, the agent who sells many fewer homes will still make more money. 
All home sales are not the same, just as all football players are not the same. The units 
used in the two examples are football players and homes. These units are not as clearly 
defined as feet or pounds and thus should only be used for comparisons with careful 


awareness of their shortcomings. 
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Measures of quantity are best used when the units are well defined, well 


understood, and well accepted by those using the measure. 


2. Efficiency 
Efficiency has been defined as "the ratio of service quantity output to the 
amount of input required to produce it [Ref. 11]." Calculation of an efficiency 
measure requires a measure of output and a measure of input. An output measure is the 
quantity of service units or product units that result from a task. An input measure is 


the quantity of a resource that has been applied to a task. 


3. Effectiveness 

Effectiveness has been defined as "the degree to which the intended public 
purposes of a service or activity are being met [Ref.11]". It is difficult to get a 
consensus on what "effectiveness" means because its definition is so context dependent. 
Effectiveness is often confused with words like efficiency, performance, and productivity 
by imprecise users. Webster’s dictionary does not help matters by listing efficient and 
productive as synonyms for effective. 

Before appropriate measures and units can be selected to measure 
effectiveness, the struggle over defining the purpose of a service or activity must be 
resolved. Businesses have a readily accepted measure of effectiveness. It’s called profit. 
If a business doesn’t make any money, it is definitely ineffective and will soon go out 
of business. Government services and non-profit organizations have a more difficult time 


measuring effectiveness because they do not produce goods or services that are 
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exchanged in the marketplace. They have no clearly defined and well-accepted measure 
of output like the monetary profit of a business. 

The question of effectiveness is especially important in military problems 
because the military spends a lot of money doing a lot of tasks where performance is 
difficult to measure. How can the military demonstrate that it is spending money 
"effectively"? The desired output of the military is "readiness", the ability to respond 
in a timely manner to any threat to or assignment from the government. Defining 
military effectiveness and then somehow quantifying it will continue to be a challenge. 

As used in operations research practice and in this thesis, effectiveness is not 
necessarily related to efficiency and productivity. A highly efficient organization may 
be ineffectual. For example, a firm that makes slide rules with a minimum number of 
workers and small amount of resources, demonstrating high efficiency, will not be very 
effective at meeting the calculating needs of today’s high school students. They all want 
electronic calculators. Likewise, a greatly effective organization may be inefficient, 
possessing substantial room for efficiency improvement. A charitable organization might 
regularly feed the homeless in its community, but if it’s paying a world class chef high 
wages to do something volunteers might do equally as well for no pay, it could improve 


its efficiency. 


4. Productivity 
Levitan and Werneke write, 
For analysts of economic performance, productivity denotes the efficiency with 


which resources--people, tools, knowledge, and energy--are used to produce goods 
and services for the marketplace. [Ref. 12] 


17 


Once units for output and input are selected, productivity can be defined as the ratio of 
output units to input units. Usually the units of output measure how much of a good or 
service is produced. The idea is that if the same amount of output is made with less 
input, productivity is higher. The interpretation of this definition of productivity is 
limited as described by Levitan and Werneke: 
An efficiency concept of productivity, however, is narrower than the idea of 
product quality, for economic efficiency is concerned only with the output of goods 
and services and not with how well the products meet consumers’ needs or 
wants...Productivity, then, is a concept of production rather than a measurement 
of consumption or social welfare. [Ref. 12] 
They point out that an efficiency concept of productivity is primarily concerned with 
quantity of output. No consideration is given as to whether or not that output is desired 
by the customer or is the best use of the resources required to produce it. An 
effectiveness concept of productivity can be used to incorporate product quality and 
customer satisfaction in the measurement. Rather than using an output measure of 
quantity in the productivity ratio, the output measure used is a measure of effectiveness. 


The effectiveness concept of productivity is not often used because it is difficult to define 


effectiveness measures. 


C. SUMMARY 

People concerned with money and budgets are often drawn to measures of 
efficiency and productivity. However, the customer is concerned with effectiveness. Is 
the organization doing what it is supposed to be doing? Is the mission being 


accomplished? To what extent is the mission being accomplished? Results are what 
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matters. The proper time to address efficiency and productivity is after effectiveness has 
been determined. Only after it has been determined that an organization is accomplishing 


its intended mission is it worthwhile to work on improving efficiency and productivity. 


i? 


IV. THE MAINTENANCE RESOURCE MANAGEMENT SYSTEM 


This chapter will describe the Maintenance Resource Management System (MRMS) 
and discuss the measures of performance calculated by MRMS. It will comment on the 
usefulness of MRMS as an internal management tool. It will describe some dangers that 
are present when MRMS data is used to make comparisons between Intermediate 
Maintenance Activities (IMAs). Finally, historical MRMS data will be analyzed to 


determine what the data show about IMA performance. 


A. BACKGROUND 

The Maintenance Resource Management System (MRMS) is an automated work 
management system used by Navy ship Intermediate Maintenance Activities (MAs). It 
is intended "to provide an information support capability to enhance maintenance 
management within the IMA community in order to ensure the material readiness of the 
Pacific and Atlantic Fleets" [Ref. 13]. 

MRMS was selected in 1988 as the common information management system for 
all Navy ship Intermediate Maintenance Activities. MRMS is in use at all Pacific Fleet 
shore IMAs and at two afloat IMAs. MRMS will be installed aboard the remaining 
afloat IMAs by the end of 1993. 

MRMS is the most recent product in a series of systems that were specifically 


designed to measure productivity. The initial impetus for the development of a 
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maintenance management system was the Department of Defense’s increased emphasis 
on productivity in the mid nineteen-seventies. The Navy started development of 
productivity measurement systems after various Government Accounting Office reports 
criticized the absence of any way to quantify productivity at Navy maintenance facilities. 

The Maintenance Resource Management System (MRMS) provides extensive 
maintenance management capability within a geographic area. It supports the area 
maintenance manager and the individual Intermediate Maintenance Activities ashore and 
afloat within that geographic area. To accomplish this, MRMS has been designed with 
two major components, the Type Commander’s Representative (TYCOM Rep) 
Component and the Intermediate Maintenance Activity (MA) Component. 

The TYCOM Rep Component provides the area maintenance manager, also known 
as the screening authority, with the capability on-line to store, screen, assign and track 
intermediate level work that has been requested for accomplishment within his geographic 
area. The IMA Component provides the capability on-line to receive, induct, plan, order 
material for, schedule, issue and monitor the progress toward completion of the assigned 
intermediate level work.[Ref. 14] 

The performance measures examined in this chapter are calculated by the IMA 
Component of the Maintenance Resource Management System. 

The centerpiece of the MRMS IMA Component is a standards-based automated 
work planning system which develops output measures for work accomplishment. Using 
Various work measurement techniques, a database of standard times for work tasks has 


been developed. The database is used by IMA planners to "build up" jobs from 
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individual work tasks. After adding time for such things as travel, job set-up, and 
worker skill level, each job is assigned a "planned manhour" value. The planned 
manhours can be thought of as the potential "worth" or output value of the job. As the 
job is completed the planned manhours are converted directly to "earned manhours", the 
unit used to measure output in MRMS. An earned manhour is thus a planned manhour 
that has been completed. Earned manhours are the only measure of output used to 
calculate performance measures in the Maintenance Resource Management System. 
Earned manhours are not the same thing as actual manhours expended on a job. The 
actual manhours can be greater or less than the earned manhours. Earned manhours can 
never exceed the number of planned manhours. Only in an ideal situation with perfect 
planning and perfect execution and no variation will earned manhours and actual 
expended manhours be equal. 


The developers of MRMS state that the system allows maintenance managers to: 


® Capture, address and dispose of more customer maintenance items than heretofore, 
@ Provide efficient area work brokering, 

@ Improve IMA shop loading, 

@ Reduce IMA shop throughput time, 

@ Promote higher quality output and reduce paperwork, and 


@ Accurately measure IMA output and input, hence performance and productivity. 
[Ref. 15] 
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This thesis primarily addresses the final item on the list, that is, performance and 
productivity measurement. 

The fact that MRMS does not measure IMA effectiveness but rather IMA 
productivity was clearly understood by the developers of the system. Biher and Eldred 
write, 

For many years the productive efficiency of these activities was not of great 
concern to the Navy. The sailor workforce was a "free good" and the overriding 
goal of ship readiness excused excesses. The IMAs were measured as to their 
effectiveness. Did they get the job done and done quickly? Quality of work and 
economy of resources were not measured. [Ref. 16] 
They clearly distinguish between measuring "quality of work and economy of resources", 
which MRMS is designed to do, and measuring effectiveness, defined by them as timely 
response, for which MRMS is not designed. 

The Maintenance Resource Management System was designed using industrial 
engineering ideas and therefore uses industrial engineering terms. Proper understanding 
of the definitions used in the system is critical. Interpreting the MRMS measurements 
using "everyday" definitions of the words used can give misleading results. 


The next section will define and describe six MRMS measures: performance, 


utilization, productivity, workload performance, load ratio, and production support ratio. 


B. MEASURES OF PERFORMANCE USED 
The following definitions are from the MRMS Methodology and Procedures 


Manual.[Ref. 17] The first section defines the quantities used to calculate the 
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MRMS performance measures. The second section defines the formulas used to calculate 


the measures. 


1. Definition of Terms 

EARNED MAN-HOURS The total number of planned man-hours assigned 
to that work which was completed by Repair Department production shops during the 
reported period. 

TOTAL MAN-HOURS ASSIGNED The total number of man-hours 
assigned to the Repair Department production shops for the reported period. 

GROSS PRODUCTIVE AVAILABLE MAN-HOURS The total number 
of man-hours assigned less the total number of productive support man-hours assigned 
for the reported period. 

GROSS PRODUCTIVE SUPPORT MAN-HOURS The total number of 
productive support man-hours assigned for the reporting period. 

NET PRODUCTIVE AVAILABLE MAN-HOURS The total number of 
productive man-hours available to production supervisors, equal to GROSS 
PRODUCTIVE AVAILABLE MAN-HOURS minus allowed deductions. 

NET PRODUCTIVE SUPPORT MAN-HOURS The total number of 
productive support man-hours available for the reporting period. 

MAN-HOURS UNASSIGNED TO JCNS The total man-hours of time when 
technicians, otherwise available for work, are not so assigned because of lack of JCNs 


(job control numbers, i.e., work requests) to be worked. 
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The following relationships between terms are true. 


TOTAL MHRS ASSIGNED = GROSS PRODUCTIVE AVAILABLE MHRS + 


GROSS PRODUCTIVE SUPPORT MHRS. 


NET PRODUCTIVE AVAILABLE MHRS = EARNED MBHRS + MBHRS 
UNASSIGNED TO JCNs + explained work delays + difference between mhrs 


expended on jobs and mhrs planned for jobs. 


Note that the last term on the right hand side of the above equation can be 
positive or negative. The details of the last two terms in the final equation are not 
presented in this thesis because they are beyond its scope. Complete information is 


available in the Maintenance Resource Management System Users Manual, Reference 13. 


2. Definition of Measures 


The six measures calculated by MRMS are 


PERFORMANCE = EARNED MANHOURS , 
NET PRODUCTIVE AVAILABLE MANHOURS 


UTILIZATION = NET PRODUCTIVE AVAILABLE MANHOURS , 
GROSS PRODUCTIVE AVAILABLE MANHOURS 
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EARNED MANHOURS 
V Ty  ————————  —  — — ——————————— s 
ne ae GROSS PRODUCTIVE AVAILABLE MANHOURS 


NET PROD AVAIL MHRS - MHRS UNASSIGNED TO JCN 


al Scalia NET PRODUCTIVE AVAILABLE MANHOURS 
WORKLOAD 
EARNED MANHOURS 
FER FORMANCE Hp 
NET PROD MHRS AVAIL - MHRS UNASSIGNED TO JCN 
PRODUCTIVE 


GROSS PRODUCTIVE SUPPORT AVAILABLE MHRS 


SUPPORT KAIIO = 
GROSS AVAILABLE MANHOURS 


Note that PRODUCTIVITY and WORKLOAD PERFORMANCE can be calculated 


directly from the other terms. That is, 


PRODUCTIVITY = PERFORMANCE x UTILIZATION , and 


PERFORMANCE 


WORKLOAD PERFORMANCE = ————————— 
LOAD RATIO 


C. ANALYSIS OF MRMS MEASURES OF PERFORMANCE 
The training materials and user’s manuals provided by the MRMS developers go 


into great detail about how to use the system to evaluate IMA workcenter performance 
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and improve the management of work. MRMS has the potential to be an excellent tool 
for the IMA manager to monitor and improve performance inside his own IMA. 
However, MRMS data is not well suited for comparisons between different IMAs. 

If MRMS data is used to compare Intermediate Maintenance Activities the 


following facts must be considered in order to avoid erroneous conclusions. 


@ No common guidance for system use is established. 

@ No performance goals are established. 

@ Aggregation of entire IMA masks overloads and underloads in individual shops. 
@ The system measures are percentages, not absolute measures. 

@ There is no link to impact on fleet readiness. 

@ Quality of work and customer satisfaction are not considered. 


@ The terms used in MRMS are easily misunderstood by an observer unfamiliar with 
the system. 


@ The Intermediate Maintenance Activities differ significantly and operate in different 


circumstances, serving varied customer bases with different impediments to good 
performance. 


D. ANALYSIS OF HISTORICAL DATA 

The concerns expressed in the preceding section explain why it can be dangerous 
to use Maintenance Resource Management System (MRMS) data to compare Intermediate 
Maintenance Activity (IMA) performance. However, there is information to be gained 


from a cautious examination of the data. The following analysis applies some statistical 
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techniques to demonstrate that there are statistical reasons for not using MRMS as a 


comparison method as well as qualitative reasons. 


1. Data Description 

The data analyzed in this section is taken from twenty-seven months of 
Maintenance Resource Management System monthly reports from four Shore 
Intermediate Maintenance Activities (SIMAs). The four SIMAs are SIMA Long Beach, 
SIMA Pearl Harbor, SIMA San Diego, and SIMA San Francisco. The months 
considered are January 1990 through March 1992. MRMS data from Tenders and Repair 
Ships was available for only one ship because not all ships have been equipped with 
MRMS. Because the afloat IMAs are so different from the shore IMAs, the data from 
the one ship was not included in the analysis. 

Table 3 shows some of the descriptive statistics for the six MRMS measures 
for each of the four shore IMAs analyzed. The means and standard deviations were 
calculated using the 27 observations of monthly measures. The original numerator and 
denominator values were not used because they were not available. Because the monthly 
measures are ratios, this approach can lead to errors under some circumstances. For 


example, consider UTILIZATION. Suppose three months data is as follows: 


month Gross Prod. Net Prod. UTILIZATION 
Avail. MHRS Avail. MHRS 
l 1000 250 0.25 
wi. 100 50 0.50 
3 100 75 0.75%. 


The actual average UTILIZATION for the three months, calculated using the sum of the 


numerators divided by the sum of the denominators, is 375/1200 = 0.31. The mean 
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TABLE 3 


SUMMARY OF MRMS PERFORMANCE MEASURES 
EXPRESSED AS PERCENTAGES 


a de ee 
meronmancs [tmssoa | ws | o2 |» | 0 | 
Treat vntor | aia | sa [90 | 7 | 107 
Sees eee eC 
vumizaTion | Long Beach | 60.6 | 25 | ot | se | ss 
[san Diego | 70s | sa | ose | at 
[san Francioo | sao | 67 | 59 | 4s | 2 
propuctivity | tongBeach | as.7_ | 48 | as || so 
= SS SS ae 
San Francisco 45.8 
LOAD RATIO ae 
Feet nator | oso | aa | or | | 0 
[sandiego | _ 96.9 | 14 | 97 | 9s | 99 
WORKLOAD | Long Beach | t0.6 | 69 | 2 | 69 | ot 
a 
sin Diego | 623 | 43 | os | as | os 
jsinFranisco | 90.0 | 74 | oo | 2 | 97 
PRODUCTIVE 
SUPPORT 
RATIO 
sin Diego | aa | ns | 2 || as 
jSanFrancico | ano | us | 40 | 39 | as 








Jie] 


UTILIZATION using the three monthly UTILIZATION values is 

(0.25+0.50+0.75)/3 = 0.50. With consideration given to this potential error, the 
second method was used to calculate the means and standard deviations in Table 3. As 
long as the denominator values in the measures remain fairly constant, the error from 
using the monthly measures instead of the original numerator and denominator values is 


small. 


2. Comparisons Between Intermediate Maintenance Activities 

Before any comparisons between the IMAs were made, the samples were 
tested for randomness using the Runs Test [Ref. 18]. All samples failed the 
test, indicating that the monthly measures cannot be tested using methods that require an 
assumption of randomly selected observations. This is assumed to occur due to the time- 
series nature of the data. Nonparametric statistical tests were therefore selected. 

The Friedman Test with Blocking was chosen to compare the samples.[Ref. 
6] To compare the four IMAs, months were the blocks and each IMA was considered 
a treatment. Because the test ranks observations only within each month and not between 


months, correlation from month to month should not affect the validity of the test results. 
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The hypotheses for the Friedman Test are 
ie Onan -O,, ane 


Pee Oe tteOrmarle coud, 


where 9, is the median of sample 1. 


The Friedman test statistic is defined as 





k 
T =|? _3 R?|-3m(k+1), where 
bk(k+1) j=1 


b is the number of blocks, 
k is the number of treatments, and 
R; is the sum of the ranks assigned to treatment j. 
Table 4 contains the test results when all four IMAs are compared. 
The decision rule for the Friedman Test is to reject the null hypothesis at the 
level a if the test statistic T exceeds the 1-a quantile of a chi-square random variable 


with k-1 degrees of freedom. 
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TABLE 4 


RESULTS OF FRIEDMAN TEST 
FOR COMPARING IMAs 
FOUR AT A TIME 


a 


WORKLOAD 67.50 reject Hy 
PERFORMANCE 

PRODUCTIVE 6G; 50 reject Hy 
SUPPORT RATIO 


Decision rule: For a = 0.05, reject H, when T > 7.815. 
















All of the tests reject the null hypothesis that all four sample medians are the 
same. That is, there are statistically significant differences between the four IMAs for 
all six performance measures. Further comparisons between the IMAs, three at a time 
and two at a time, give the results shown in Tables 5 and 6. In several categories, it is 
not possible to say with confidence that there are differences in the measures at the four 
IMAs. Table 7 summarizes the findings. 


It would appear that Pearl Harbor was the "best" IMA of the four because 


it has the highest numbers in two of the six measures, and tied for best with San Diego 
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and San Francisco in two others. However, if UTILIZATION was deemed to be the 
most significant measure, San Diego would be the "best" IMA even though it has the 
lowest scores in several other measures. 

All conclusions about relative performance are suspect without a common 
directive for implementation of the Maintenance Resource Management System. 
Currently, each Intermediate Maintenance Activity decides how it will count production 
support personnel and direct production personnel. They also decide independently about 
applying allowed deductions. For example, some IMAs consider the productive workday 


to be eight hours long, while others allow a thirty minute administrative deduction for 


TABLE 5 


RESULTS OF FRIEDMAN TEST 
FOR COMPARING IMAs THREE AT A TIME 


LB=1, PH=2, SD=3, SF=4 


nn EE DR 
PERFORMANCE 50.30 30.91 37.85 49.13 
UTILIZATION 41.06 39.5 


PRODUCTIVITY 19.91 16.08 29.17 
LOAD RATIO 25.9) 47.91 43.67 41.17 


WORKLOAD 
PERFORMANCE 


PRODUCTIVE 48.22 45.39 43.17 34.06 
SUPPORT RATIO 


Decision rule: For a = 0.05, reject Hy) when T > 5.99. 
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TABLE 6 


RESULTS OF FRIEDMAN TEST 
FOR COMPARING IMAs TWO AT A TIME 


PERFORMANCE LOAD 
RATIO 







PRODUCTIVE : 


SUPPORT 
RATIO 






WORKLOAD 
PERFORMANCE 






moming muster and afternoon cleanup, giving a seven-and-a-half hour workday. Over 
a period of months and hundreds of workers, that thirty minute difference may show 
performance differences that are not real, but rather a result of the definition of MRMS 
measures policies. The MRMS measures may indeed indicate performance differences, 


but the users of MRMS might be measuring different things. 


E. SUMMARY 

Maintenance Resource Management System performance data should be used 
internally by each Intermediate Maintenance Activity as a management tool to monitor 
and improve efficiency. After all, the system was designed to do just that. 

Officials at Commander, Naval Surface Force, Pacific (COMNAVSURFPAC) must 


provide standard guidance to the IMAs for use of MRMS before they can use MRMS 
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TABLE 7 


SUMMARY OF COMPARISONS 
BETWEEN IMAs 


=e 
Beach Harbor Diego Francisco 
ee 
Se 
Te ee a 


| WORKLOAD 
PERFORMANCE 






PRODUCTIVE 
SUPPORT RATIO 


performance data to compare IMAs. An instruction defining exactly how to categorize 
manhours, stating explicitly how to account for authorized deductions, and providing 
some reasonable performance goal would be helpful. In addition, they must be careful 
to understand what the data does and does not represent. MRMS data measures 
efficiency, not effectiveness. COMNAVSURFPAC officials must use other categories 
of measures together with the MRMS measurement data to get a complete picture of 


IMA performance. 
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V. MONTHLY IMA UTILIZATION REPORT 


This chapter describes the Monthly IMA Utilization Report, discusses its use, and 


examines Fiscal Year 1991 data from the report. 


A. DESCRIPTION 

The Monthly IMA Utilization Report is a report of quantities. There is nothing in 
the data, as reported, that purports to measure efficiency or effectiveness. 

The report is required by Commander, Naval Surface Force, U.S. Pacific Fleet 
(COMNAVSURFPAC).[Ref. 19] The data collected in the report are divided 
into several sections, some of which are Manning Allocation, IMA Manning Utilization, 
Man Hour Utilization, and Work Request Submission. 

The instruction establishing the report states, 

The efficient and effective utilization of maintenance resources is a high priority 
matter for the Navy. Data on utilization is an important input for monitoring IMA 
performance, TYCOM decisions on maintenance, meeting CINCPACFLT and 
CNO reporting requirements and developing budgetary justification for manpower 
and funding. [Ref. 19] 

Although the instruction includes the words "efficient" and "effective", it does not 
define what is meant by the words in the context of IMA performance. It also does not 


explain how the data collected in the report can be used to measure efficiency and 


effectiveness. 
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Note that the term "utilization" is used here to mean "usage" or "employment". 
It is not the same definition of “utilization” used in the Maintenance Resource 
Management System (MRMS) as described in Chapter IV. This dual usage of the same 
word, but with different meanings, is a source of much confusion to people using both 


the MRMS report and the Monthly IMA Utilization Report. 


B. DISCUSSION 

The numbers collected in the Monthly IMA Utilization Report answer questions of 
the form "How much?" How many manhours were expended on customers this month? 
How many work requests were completed? How many work requests were rejected? 
How many people were assigned to the Intermediate Maintenance Activity? There is an 
implied understanding that more is better in measures of quantity. This is not necessarily 
true. Doing more jobs or expending more manhours on customers is not really a goal 
of an Intermediate Maintenance Activity. 

The assumption that more is obviously better is a common mistake when defining 
and using measures of effectiveness. A classic example is that of analysts studying 
Allied antisubmarine warfare efforts against German U-boats during World War IL. 
Originally, the analysts proposed “number of U-boats sunk" as a measure of 
antisubmarine warfare effectiveness. This definition assumes that the goal of 
antisubmarine warfare was to sink U-boats. However, the real goal was to get cargo 
across the ocean without having it sunk by enemy attack. Thus, an appropriate measure 


of antisubmarine warfare effectiveness was merchant ships protected from sinking or tons 
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of cargo delivered, not number of U-boats sunk. The number of U-boats sunk might 
actually go down while cargo delivered goes up due to better antisubmarine protection 
for cargo ship convoys and less emphasis on submarine sinking. [Ref. 20] 

In a like manner, measuring the amount of work done by an Intermediate 
Maintenance Activity may not be measuring how well the IMA is achieving its 
performance goal. In theory the IMAs could work 24 hours a day making plaques and 
sheetmetal cabinets, while pumps and motors and radars and sonars lay broken. The 
IMA certainly did a lot of work, but what did it accomplish? What did it contribute to 
Fleet readiness? The IMA may fix the pumps and motors and radars and sonars, only 
to have them break again due to poor quality work. The IMA will expend more 
manhours and complete more jobs because of excessive rework. Is this what is meant 
by a "good" or "efficient" or "effective" IMA? Probably not. 

The real goal of an IMA is to do, in a timely manner, all the intermediate level 
work that needs to be done. It may be true that the reason a particular IMA is doing less 
work than another is because there is less work to be done. Perhaps all the customer 
ships in their area have excellent maintenance and nothing ever breaks. If all the work 
that needs to be done is complete, it’s foolish to "make work" just to drive up reported 
quantities. 

Likewise, an IMA may be working overtime, expending massive amounts of 
manhours on customers and completing thousands of jobs, but only half of the work that 


needs to be done is being accomplished. 
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The measures of quantity of work reported in the Monthly IMA Utilization Report 
do not address any aspect of how much work needs to be done. 

Thus, the validity and meaningfulness of the quantities reported depends on the 
integrity of the IMA management, screening authority, and customer ships. It must be 
assumed that all work that needs to be done is attempted and that only work that needs 
to be done and should be done by an IMA is attempted. Cheating, once it is clear that 
number of jobs or number of manhours expended is the "important" number, is a 
temptation for IMAs more determined to look good than to actually be good. Because 
of this, both quantities are looked at with a jaundiced eye. “Gut feeling" and "waterfront 
reputation” carry more weight as an evaluation of IMA performance than the numbers 
in the Monthly IMA Utilization Report. 

In spite of the distrust with which the numbers are viewed, they are collected and 
have the potential to provide useful information about IMA performance. The next 


section examines data reported in Fiscal Year 1991. 


C. DATA 

The data examined in this section is from the Monthly IMA Utilization Reports 
submitted in Fiscal Year 1991 by all eleven U.S. Pacific Fleet Intermediate Maintenance 
Activities (IMAs). Table 8 contains the legend used to identify the IMAs in all graphs 
and tables of performance data. 

First the data as reported will be examined. Second some derived (calculated) 


terms are examined. 
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1. Data as Reported 

The two leading ways to quantify the amount of work done by an 
Intermediate Maintenance Activity are customer manhours expended and number of work 
requests completed. Quantities for Fiscal Year 1991 are shown in Figures 1 and 2. Both 
of these measures mask aspects of what was actually accomplished by the IMA. 
Counting manhours doesn’t indicate how well the workers performed. Did they do the 
work in a "reasonable" amount of time or did they use "too much" time? What type of 
work was done? What type of work was left undone? There is a potential to overlook 
the fact that some IMAs have more personnel than others. It seems obvious that an IMA 


with more workers would expend more manhours on customers. 


TABLE 8 


LEGEND USED IN GRAPHS AND TABLES 


USS Acadia Puget Sound 
(AD-42) 
USS Cape Cod NAB NAB Coronado : 
(AD-43) | 
GOM USS Samuel Gompers SIMA Long Beach 
(AD-37) 
JA USS Jason PH SIMA Pearl Harbor 
(AR-8) 
111 SD SIMA San Diego 
(AD-15) | 


SF SIMA San Francisco 
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Figure 1. Manhours Expended on Customers in FY 1991 by IMAs. 


The problem with counting work requests is that the work requests are so 
dissimilar. Unlike identical items manufactured on an assembly line, intermediate ship 
maintenance tasks are extremely diverse. A work request could require one locksmith 
to expend two hours drilling open a locked file cabinet or it could require several 
electricians and welders to work many days installing waterline security lights. A job 
repairing a main feed pump on a Sunday afternoon might allow a ship to meet a critical 
operational commitment on Monday morning, while a job to manufacture decorative 
brow skirts could require hours and hours of labor, but have no discernible impact on 


readiness. The difficulty and significance of work requests can differ greatly. In spite 
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Figure 2. Number of work requests completed in FY 1991 by IMAs. 


of this, each work request described above would still count as one item, even though 
it could be argued that their worth is not equal. 

Absolute quantities give no information about efficiency or effectiveness. All 
that can be concluded is that one IMA expended more manhours or completed more work 
requests than another. Further examination of various derived values may be better 


suited to answer questions of efficiency and effectiveness. 


2. Derived (Calculated) Values 
There isn’t anything in the Monthly IMA Utilization Report that can 


reasonably be used to measure efficiency. However, various numbers can be calculated 
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to describe IMA performance and that performance can be deemed more or less 


effective. It all depends on how "effective" is defined. 


a. Percent of Work Requests Accepted 
If an effective IMA is defined as one that accepts as many work 
requests as are presented to it, then “percent of work requests accepted" is an appropriate 
measure of effectiveness (MOE). A higher value would indicate a more effective IMA. 


Table 9 lists the percent of work requests accepted by the IMAs in Fiscal Year 1991. 


TABLE 9 


WORK REQUEST ACCEPTANCE RATE 


# wr # wr reject % accept % reject 
accept 
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Work requests can be rejected for good reasons which may not be 
caused by a performance deficiency by the IMA. For example, a work request may be 
submitted that is beyond the capability of a particular IMA. The customer ships may be 
filling out the work requests improperly, providing inaccurate or insufficient information 
making it impossible for the IMA to accept the work. A piece of equipment may no 
longer be supported by its manufacturer, making repair parts unavailable. Thus, some 
percentage of work request rejection is acceptable. Indeed, the 10% rejection rate at 
SIMA Puget Sound and NAB Coronado may be caused in part by capability shortcomings 
of their extremely limited facilities. Big changes in the MOE at an IMA or an IMA 
whose MOE differs greatly from that of the other IMAs should motivate further 
investigation as to why that occurred. Again, the importance of using measurement data 
as a way to improve performance is emphasized. The reasons for changes in the MOEs 
must be sought out and either praised or corrected, else the effort expended in 


measurement is wasted. 


b. When Work Requests are Accepted 
COMNAVSURFPAC divides work requests accepted into three groups 
based on when they are accepted: advance, supplemental, and late or emergent. For 
Tenders and Repair ships advance is defined as 35 days prior to availability start date. 
For Shore IMAs, advance is defined as 45 days prior to availability start date. 
Supplemental work requests are those accepted between the advance date and the 


availability start date. Late or emergent work requests are work requests received after 


Ad 





‘the availability start date and all work requests received that are not associated with an 
availability. 

Table 10 shows the distribution of work requests accepted by the IMAs 
‘nto the three categories. Interpreting the breakdown as an indication of IMA 
effectiveness can lead to completely opposite findings. Consider the percentage of late 
_ work requests accepted. A high value here could indicate a good performance trait or 
a bad performance trait. The good performance trait indicated is a willingness to accept 
emergent work and/or aggressive ‘dentification of work on the customer ships even after 
the availability start date. The bad performance trait indicated is slow response to work 
requests submitted, delaying the work acceptance decision, and/or a failure on the part 
of the customer ship to plan ahead and identify work in advance. 

A high number of late work requests might result due to short notice 
availabilities, scheduled with little lead time. Such availabilities are common for 
deployed Tenders and Repair Ships. 

The analysis of when work requests were accepted does describe an 
aspect of IMA performance, but additional information as to why the particular 
distribution occurred is necessary before a given distribution can be characterized as 


indicating "good" or "bad" performance. 


c.  Manhours per Work Request 
The two quantities of output measured, customer manhours and number 


of work requests, can be combined to calculate manhours per work request. If all IMAs 
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TABLE 10 
ACCEPTANCE TIMING OF 


WORK REQUESTS ACCEPTED 
FY 1991 


# wr 
accepted pH — = a 





are doing approximately the same mix of simple and difficult tasks, the number should 
be about the same for all IMAs. Table 11 lists manhours per work request for the IMAs. 
USS Prairie (AD-15) at 172.4 and USS Cape Cod (AD-43) at 23.1 stand out as different 
among the tenders, and NAB Coronado at 144.6 and SIMA Puget Sound at 15.9 stand 
out as different among the shore facilities. The numbers indicate that the work requests 
completed by USS Prairie and NAB Coronado required more manhours, on average, than 


those completed by SIMA Puget Sound and NAB Coronado. Higher manhours per work 
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request could indicate more complex jobs requiring more time or people to complete, 
easy jobs that are just large, or overstaffing on jobs. Lower manhours per work request 
could indicate more efficient workmanship, or an unwillingness to accept difficult jobs 
that need many manhours to complete. The determination as to whether or not the 
significantly different values for manhours per work request indicate a good performance 


trait or a bad performance trait can only be made after further investigation. 











TABLE 11 
DERIVED MEASURES USING MANHOURS 
FY 1991 
mhrs/work mhrs/ima person/month | mhrs/ repair labor 
request person/month 
Tac [us | «| 9 
| NE c83 ee ef _ 1955 


GOM 
a ee 
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Table 11 also contains values for customer manhours per person 
assigned to the IMA per month and customer manhours per person assigned to repair 
labor per month. The first includes all IMA personnel, including both repair labor and 
repair support. The second uses only personnel assigned to direct repair production 
work. Again, a few observations stand out as noticeably different from the others. USS 
Cape Cod (AD-43) has the smallest numbers of customer manhours expended per person 
per month in both columns, 23.3 and 42.4. USS Samuel Gompers (AD-37) has the 
largest numbers, 98.3 and 195.5. The USS Samuel Gompers numbers are more than 
four times the numbers reported by USS Cape Cod. This large difference indicates that 


the two ships are probably defining their units of measurement differently. 


d. Relation Between Number of Personnel and Amount of Work Done 

It seems reasonable to examine the relationship between the amount of 

repair labor available to do work and the number of customer manhours expended. More 
workers should lead to more work completed. Figure 4 shows a scatter plot of customer 
manhours expended per year versus repair labor man-months per year. Superimposed 
on the plot is a line fitted using a least squares fit. The commercial software MINITAB 


was used to do the least squares fit. 
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Relation Between Repair Labor and Customer Manhours 
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Figure 3. Plot of customer manhours expended vs. repair labor man-months. Each 
point represents one IMA. 
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The eleven data points, one for each IMA, indicate a strong linear 
relationship between direct production repair labor man-months and customer manhours 


expended. The p-value of the fit is 0.000. The fitted equation is 


customer manhours expended per year = 


97.59 x repair man-months per year. 


This means that on average, each person assigned to direct production repair labor, not 
production support labor, expends 97.59 labor hours per month on customers. Obviously 
some individuals will work more than others due to the different types of work done in 
the various shops. However, the fitted equation does immediately present the question 
of what was done during all the other available workhours of the month. Assuming a 
work month of twenty 7.5-hour days leaves over fifty hours per month per worker that 
are not charged to customer jobs. Perhaps that time is required to fulfill training, 
internal maintenance, and other administrative needs. Perhaps the time is available for 
additional work. Additional study is necessary to determine if the 97.59 labor hours per 
month per worker is the appropriate performance level for the IMAs. 

There are two observations on the plot that are noticeable as having 
large residuals. The one below the line represents USS Cape Cod (AD-43) and the one 
above the line represents USS Samuel Gompers (AD-37). Some of the apparent disparity 
in performance may represent plain-old better service, although such a huge difference 


seems unlikely. Another explanation for the results can be found in the Fiscal Year 1991 
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operating schedules of the two ships. USS Cape Cod was deployed to the Persian Gulf 
from February 1991 to May 1991, during Operation Desert Storm. Due to the nature 
of the operations in the Persian Gulf, ships were not made available for routine work 
availabilities and thus USS Cape Cod did not expend the usual number of labor hours on 
customers. Also, USS Cape Cod was underway for 93 days of the year, an unusually 
high number for a tender. USS Samuel Gompers, on the other hand, arrived in the 
Persian Gulf in May 1991, just as USS Cape Cod was returning to San Diego, and 
remained until August 1991. Many of the ships that had postponed routine intermediate 
maintenance availabilities were now made available for work and USS Samuel Gompers 
had more than the usual amount of work requests to accomplish. USS Samuel Gompers 
also had fewer underway days than USS Cape Cod, accumulating only 72 underway days 
during the year. 

This explanation of the performance differences of the USS Cape Cod 
and USS Samuel Gompers is a good example of how measurement data should be used. 
Measurements from all eleven IMAs were looked at, but only the extraordinary cases 


required additional investigation and explanation. 


S1 


VI. SUGGESTED MEASURE OF EFFECTIVENESS 


The Intermediate Maintenance Activity IMA) performance measures discussed in 
Chapters IV and V address efficiency and quantity. The measures are presented in 
existing reports or can be calculated from data in those reports. Although the 
Maintenance Resource Management System (MRMS) and the Monthly IMA Utilization 
Report provide a lot of information, there is still very little information available about 
IMA effectiveness. This chapter will propose some measures of effectiveness (MOEs) 
for Intermediate Maintenance Activities. 

Although it is most appropriate to determine measures of effectiveness first, before 
efficiency and quantity measures, effectiveness measures are presented last in this thesis. 
This is because the measures presented in this chapter are not currently collected by 
COMNAVSURFPAC officials. Measurement data from currently collected reports were 
examined first for their usefulness in measuring effectiveness. After it was determined 
that additional measures of effectiveness would be desirable, this chapter was prepared 
to propose some MOEs to augment the performance data already collected by 


COMNAVSURFPAC officials. 
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A. DISCUSSION 

The first difficulty encountered when trying to measure effectiveness is defining 
effectiveness in the context of Intermediate Maintenance Activity (IMA) performance. 
What is an effective IMA? 

Because ship maintenance is a service provided to customer ships by the 
Intermediate Maintenance Activity, it is appropriate to include the customer in the 
definition of IMA effectiveness. What is it that the ships receiving maintenance want? 
When discussing the issue of IMA effectiveness with maintenance providers and 
maintenance customers, two things are mentioned repeatedly as desirable traits for IMAs: 
high quality work and timely response to work requests. That is, an effective IMA is 
one that demonstrates willingness to do work, does the work quickly, and does the work 
well. 

Measuring quality of work is beyond the scope of this thesis. A measure of 


timeliness that captures some essence of IMA responsiveness is defined below. 


B. SELECTION OF APPROPRIATE TIME INTERVAL 

The aspect of effectiveness to be measured is timeliness. Effectiveness can be 
defined in absolute or relative terms. A relatively "more effective” IMA is defined as 
one that does more work faster than another IMA. An absolute definition of "effective" 
could be a certain percentage of jobs completed in a certain number of days. For 
example, absolute effectiveness could be defined as 70% of jobs completed in 180 days 


or less. The specific time interval used is defined below. 
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Three different time intervals were considered to represent the effectiveness of 
Intermediate Maintenance Activities. Two were discarded as inappropriate. The third 
was selected even though it has some drawbacks. 

The first time interval considered was the time between when an IMA started a job 
and when the IMA finished the job. This would measure something primarily within the 
control of the IMA but has the same problem of merely counting the number of work 
requests completed. It only considers the work the IMA does, not the work it chooses 
not to do. It doesn’t take into account that the customer ship may have been waiting for 
many months to get an IMA to say yes to the job, and then to finally start the job. 

The second time interval considered was the time that a job stays on a ship’s 
worklist. The time interval would be measured from the date the ship’s personnel put 
the work request on the worklist to the date of completion of the work request by the 
IMA workers. The mean-time-on-the-job-list would be calculated for all work requests 
for each ship. This does assume that the jobs eventually get done because the time 
interval cannot be calculated until the job is completed. If the ships in a particular 
homeport had shorter mean-time-on-the-job-list, the IMAs in that port would be defined 
as more effective than IMAs in other ports. This idea won’t work as a measure of IMA 
effectiveness because it doesn’t identify individual Intermediate Maintenance Activity 
performance. Each ship could be served by many different IMAs and so the mean-time- 
on-the-job-list for a particular ship would be more a measure of the individual ship’s 


effectiveness in using the maintenance system than a measure of IMA effectiveness. 
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The second time interval described above can be used as a measure of IMA 
effectiveness if the work requests are grouped by which IMA completes the work 
requests rather than by which ship submitted the work requests. The time interval is 
measured from the date a work request is put on the ship’s worklist to the date the work 
request is completed by IMA personnel. The time interval will be named A and is used 
to calculate two measures of effectiveness (MOEs) described as follows. 

First select the group of work requests to be used to calculate the MOEs. The 
group can be all the work requests completed by an IMA in any given time period, for 
example, a quarter or a year or a period of several years. Next determine A for each 
work request where 

A = date completed - date identified by ship. 
The measures of effectiveness are 
» = humber of days, A, such that p percent of work requests in the group 
considered have A < T,, and 
P, = percent of work requests in the group considered that have A § t. 


The two measures of effectiveness are related to one another as follows: 


Pir) = x, and 
Tp) = xX. 


For example, consider T,, = 25. It follows from the definition that P,, = 10. The 


percentage of jobs completed in 25 days or less, P,;, is 10. The number of days required 
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to complete 10% of the jobs, T,), is 25. Next consider P,, = 12.8. It follows from the 
definition that T,,, = 30. The number of days to complete 12.8% of the jobs, T,,, is 
30. The percentage of jobs completed in 30 days or less, P39, is 12.8. Defining both 
T, and P, allows the user to emphasize one of two things by selecting the most 
appropriate measure. Using P,, the percentage of work completed in a certain time t is 
emphasized. If it is desired to emphasize how long it takes to do a certain percentage 
of work p, T, can be used. 

These are compromise measures of effectiveness that have good points and bad 
points. The good point is that the measures do capture the essence of timely response 
to customer ship maintenance needs. It measures how long the ship is waiting to get 
something done. 

The bad point about the timeliness MOEs is that not all of the time interval A is 
under the control of the IMA. Both T, and P, depend on how well the maintenance 
system as a whole is working. Part of the wait may be unavoidable. The ship 
continually identifies problems and places work requests on its worklist even if it is 
deployed and knows it will be several weeks before it has a maintenance availability. 


Some other factors influencing the length of the time interval A are: 


@ Accuracy of information provided by ship, 
@ Responsiveness of screening authority to task IMA, 
@ Speed of planning department at IMA, 


@ Making ship available for work (schedule), 
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® Doing the work, and 


® Completing paperwork. 


It is important to remember that some portion of these delays is an acceptable and 
understood part of the time interval A. An MOE doesn’t have to be a "perfect", just 
"appropriate". An MOE is not designed to assign blame or accuse an IMA of dereliction 
of duty. Rather, it must capture enough of the trait of interest to be helpful in 
monitoring performance and identifying potential improvements. If numbers come out 
other than expected, or there is something odd, further investigation is indicated. 
Remember, measures of performance should be used to identify departure from the 
acceptable norm. They are supposed to be a method to indicate ways for performance 
improvement. 

A measure of effectiveness is something that should change if additional resources 
are added, demands are reduced, or procedures are improved. T, and P, meet these 


criteria. 


C. RESULTS 

The two measures of effectiveness, T,, and P,, were calculated using data provided 
by the Naval Sea Logistics Center. The necessary data about all work requests 
completed by the IMAs in 1990 and 1991 were extracted from the Maintenance Material 
Management (3M) System Central Data Bank. Data for jobs completed by SIMA Puget 


Sound, NAB Coronado, and SIMA San Francisco were not available. The time interval 
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A was calculated for each job. Then T, and P, were calculated treating the two years of 
data as one group. Tables 12 and 13 display the results. 
TABLE 12 


MEASURE OF EFFECTIVENESS T, 
FOR INTERMEDIATE MAINTENANCE ACTIVITIES 
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TABLE 13 


MEASURE OF EFFECTIVENESS P, 
FOR INTERMEDIATE MAINTENANCE ACTIVITIES 


__ FE ee 
| Py | 33.4 [46.3 | 57.8 | 485 | 48.4 | 25.2 | 39.4 | 43.8 

mainGes .|.720 | 704 | 763 | 73.1 | 515.1 663 | 713° 
Pwo |] 89.4 | 91.7] 93.6 | 92.2 | 90.7 | 79.9 | 86.4 | 90.7_ 








1. Comparison of Empirical Distribution Functions of A 

Before individual values of T,, and P, are examined, the question of statistical 
significance of the differences in those measures is addressed. Both measures describe 
the empirical distribution function (EDF) of the time intervals A for a particular 
Intermediate Maintenance Activity. The measures will differ in a statistically significant 
way only if the empirical distribution functions are different. The Kolmogorov-Smirmov 
(K-S) Two-Sample Test was used to compare the EDFs [Ref. 18]. At significance level 
0.05, the K-S test rejects the null hypothesis that the EDFs are the same in 24 of 28 
possible combinations of two IMAs. This means that in almost all cases the empirical 
distribution functions of the As do differ significantly from one another, indicating that 


differences in T, and P, are statistically significant. The cases where the null hypothesis 
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can not be rejected are USS Acadia/SIMA Pearl Harbor, USS Jason/USS Cape Cod, USS 


Prairie/USS Cape Cod, and SIMA San Diego/USS Cape Cod. 


2. Discussion of T,, 

The USS Samuel Gompers (AD-37) has the smallest values of T, and SIMA 
Long Beach has the largest values of T,. Ts) for USS Samuel Gompers is 72 and T,, for 
SIMA Long Beach is 173. This means that 50% of the work requests completed by USS 
Samuel Gompers were identified by the customer ships less than or equal to 72 days 
prior to completion, and 50% of the work requests completed by SIMA Long Beach were 
identified less than or equal to 173 days prior to completion. 

The existence of the difference, although not its magnitude, makes sense 
because of the differences in how SIMAs and Tenders operate. Tenders are expected to 
react to short notice work requests, especially while deployed. Often when the tenders 
are deployed, they will be the only repair facility in the area and may have few 
customers to service. The tender may ask the customer ships to identify additional work 
during the availabilities to more fully load the repair workcenters. 

The SIMAs, in contrast, are more dependent on long-term planning and 
scheduling of work. As fixed facilities serving many customer ships, they are often 
unable to accept additional work identified after the availability work package has been 
specified. Because the work packages must be submitted several weeks in advance, it 


is not unreasonable that the SIMAs have higher values for T,. 
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Four of the tenders show similar values of T,, all lower than the T, values 
of the SIMAs. USS Acadia is an exception, showing T, values that look more like the 
SIMA values. 

Again, this MOE is merely an indicator of performance. The MOE itself 
does not explain why the differences between IMAs occur. Values that stand out from 


the others must be investigated further to be explained. 


3. Discussion of P, 

Another way of examining the timeliness of work request completion by 
Intermediate Maintenance Activities is P,. Current Pacific Fleet policy is to schedule 
each ship for an intermediate maintenance availability once per quarter. If many jobs 
have A > 90 days, that is, Po) is small, it indicates that either ships are not being 
scheduled for availabilities quarterly, as desired, and/or the Intermediate Maintenance 
Activities are not completing all of the outstanding jobs during the availabilities. Thus, 
many work requests are carried on the ships’ worklists from one availability to another 
until finally there is an opportunity for an IMA to complete the work request. 

Note that SIMA Long Beach has the lowest values for P, and USS Samuel 
Gompers has the highest values for P,. P,, for USS Samuel Gompers is 57.8 and P,, for 
SIMA Long Beach is 25.2. Only 25.2% of the work requests completed by SIMA Long 
Beach were identified less than or equal to 90 days prior to completion, while 57.8% is 
the proportion for USS Samuel Gompers. 

Since no IMA has a P,, greater than 57.8, it appears that a large number of 


work requests linger on ships’ worklists for long periods of time. Some of the time 
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delay may be due to ship operating schedules, backlog of work requests caused by 
insufficient IMA capacity, or failure of ships’ personnel to properly use the maintenance 
system. Again, The measure of effectiveness, P,, is only an indicator of performance. 
Explanations as to why the values are what they are must be discovered through further 


investigation. 


D. SUMMARY 

The two measures of effectiveness, T, and P,, provide information about the 
timeliness of Intermediate Maintenance Activity response to work requests. They can 
be used to identify and correct impediments to a faster turnaround time for completion 
of work requests by focusing the attention of COMNAVSURFPAC officials on the IMAs 
that are performing poorly. Collected over several periods of time, T, and P, can be 


used to monitor trends in performance. 
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VII. CONCLUSIONS AND RECOMMENDATIONS 


This thesis has examined various performance measures for ship Intermediate 
Maintenance Activities. Performance measures are currently collected for quantity, 
efficiency, and productivity. Effectiveness is not currently measured, but can be 
measured as recommended in Chapter VI. All of the measures provide some information 
about IMA performance and can be used by Naval Surface Force Pacific officials to 
monitor IMA performance. However, two things must be carefully considered by users 
of the performance measures to avoid erroneous conclusions. 

First, understanding the definitions of what is being measured and what units are 
being used is critical. Second, statistical tests are necessary to say with confidence that 
the performance of IMAs differ. Many of the differences in performance measures are 
not statistically significant. 

No one performance measure can quantify all aspects of IMA performance. Even 
all of them together do not tell the whole story of IMA performance. As Moore warns, 
Beware the arrogance that says that everything can be measured, or that only things 
we Can measure are important. The world contains much that is beyond the grasp 

of statistics. [Ref. 10] 
There are intangibles of Intermediate Maintenance Activity performance that will 
augment measurement data when IMA performance is evaluated. 

Recommendations for further study can be grouped as additional research in 


support of existing performance measures, and as additional ideas for measuring effectiveness. 
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It was shown in Chapter V that, on average, 97.59 manhours were expended on 
customers per direct repair production worker per month. It is unknown if this is a 
reasonable or acceptable amount. A study should be done to discover how many 
manhours per worker are actually available for productive work after all mandatory Navy 
programs are completed. Such things as mandatory safety training, career counseling, 
equipment maintenance, and other duties reduce the number of manhours available to 
expend on customers. Perhaps the 97.59 customer manhours is close to what is actually 
available to expend on customers. 

More ways to measure effectiveness, in addition to T, and P, recommended in 
Chapter VI, should be developed. A potentially successful method is a customer 
satisfaction survey. A scientifically designed and analyzed survey of customer ships, 
taken at regular intervals, would give valuable information about the effectiveness of IMA 
performance. 

Both proper use of measurement data and concern for customer satisfaction are 
elements of Total Quality Management (TQM), a management philosophy being 
embraced by the United States Navy. One author summarizes the key elements of TQM 
as follows: 

The pillars of the cultural change are: focus on customers (both internal and 


external); data-driven continuous improvement; and new ways to involve employees 
and management in joint identification and solution of problems. [Ref. 21] 
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Measurement pervades Total Quality Management and is critical to its success. The 
discussion presented in this thesis points out facts and concerns about performance 
measurement to be addressed whether or not TQM is implemented at ship Intermediate 


Maintenance Activities. 
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